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Abstract 

Inspired by the results of Baik, Deift and Johansson on the limiting distribution 
of the lengths of the longest increasing subsequences in random permutations, we 
find those limiting distributions for pattern-restricted permutations in which the 
pattern is any one of the six patterns of length 3. We show that the (132)-avoiding 
case is identical to the distribution of heights of ordered trees, and that the (321)- 
avoiding case has interesting connections with a well known theorem of Erdos- 
Szekeres. 



1 Introduction 

A great deal of spectacular work has been done on determining the distribution of the 
length of the longest increasing subsequence, LIS(p), in a random permutation p. This 
work culminated in the paper of Baik, Deift and Johansson |2I which found the limiting 
distribution function. 

Historically, the average length, /i(n), of the LIS had been found to be > y/n/2 P 
and then > 2^/n ^2)- The correct order of magnitude was established in |[18j, where it 
was shown to be ~ 2^/n. Next the standard deviation (T{n) had been estimated jlH] to 
be approximately n^^^ but was not proved to be so. Finally, in fS], the complete limiting 
distribution function of the normalized random variable (LIS— yu(n))/n^/^ was determined. 

We can ask the same questions in some given subset of the set of all permutations. 
For example, what is the distribution of the length of the longest increasing subsequence 
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in a randomly chosen permutation from the set of those that avoid the pattern^ (231), 
or the pattern (321)? The tools are available for the study of such problems. A paper of 
Reifegerste P3! finds a simple expression for the frequencies of different LIS lengths in the 
(231)-avoiding case. For (132)-avoiding permutations it turns out that the distributions 
are identical with known distribution functions of ordered trees by height, so we can 
simply quote the known results. Finally, for (321)-avoiding permutations the problem is 
related to a celebrated theorem of Erdos-Szekeres. In this case we exhibit the relevant 
statistics and find explicitly the limiting distribution function. 

We remark that for all six of the patterns of length 3 it is true that the number of 
permutations of n letters that avoid the pattern is the Catalan number C„ = (^") /(n+ 1). 
So for each of the six patterns that we will study here, we will be asking what fraction 
of the roughly c4"'/n^/^ permutations that avoid that pattern have a LIS of some given 
length. 

Our main results are the following. 

Theorem 1. In the class of [231) -avoiding permutations of n letters, and in the class of 
(312) -avoiding permutations of n letters, the length of the longest increasing subsequence 
has mean (n + l)/2, and standard deviation ~ y/n/2. Moreover, the random variable 
{LIS{p) — (n + l)/2)/(y^/2) has asymptotically the standard normal distribution. 

Theorem 2. In the class of [132) -avoiding permutations of n letters, and in the class of 
{213) -avoiding permutations of n letters, the length of the longest increasing subsequence 
has mean = y/rm -\- 0{ni) and standard deviation 

= civ^ + 0(n3) (^ci = ^TT - l) = 0.38506..^ . (1.1) 

Moreover, the normalized random variable 

defined for n-permutations p that avoid the pattern (132), satisfies 

oo 

lim Prob(X„(p) <e)= y (1 - 2t\e + ^f)e-^'+^^"'" {6 > -v^). (1.2) 

n— >oo ' ^ 

i=— oo 

Theorem 3. For permutations p in the class of (321) -avoiding permutations of n letters, 
define the random variable 



Xnip) 



LIS(p) - f 



n 



permutation p avoids (132) (resp. (231), (321)) if there do not exist \ <i<i<k<n s.t. 
p{i) < p{k) < p{j) (resp. p{k) < p{i) < p{j), p{i) > p{j) > p{k)). 
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Then we have 



lim Prob(X„(j9) <Q) = — u^/^e-'^du 

n^oo Jo 



r(3/2) 

where r{z,w) is the incomplete Gamma function 



Jo 



2 Avoiding the pattern (231 



One situation where it is simple to deduce the hmiting distribution is that of (231)- 
avoiding permutations. This is because Reifegerste jHj has shown that the number of 
(231)-avoiding permutations whose longest increasing subsequence has length exactly k 
is 

1 fn\ f n 



The number of all (231)-avoiding permutations of n letters is C„, the Catalan number. 
From this it is easy to check that the mean length of the LIS in this family is (n + l)/2 
and the standard deviation is + 0(1). Finally it is a straightforward exercise in 

Stirling's formula to find that 

e(n.n±l + ^f) 2 
lim \/n — ^ — = — :=€ 



n— >oo 



Cn ^/7^ 



This says that for (231)-avoiding permutations p, the random variable 



Xip) 



LIS{p) 



n+l 
2 



has asymptotically the standard normal distribution. 

It seems noteworthy that in this case the mean is quite large, i.e., typically such a 
permutation contains an increasing subsequence whose length is about half that of the 
permutation itself. 

It also is noteworthy that ^2,.\\ shows that the number of (231)-avoiding permutations 
of n letters whose longest increasing subsequence has length k is the same as the number 
whose longest increasing subsequence has length n — k + 1. It is arresting that the number 
of (231)-avoiding permutations of, say, 1000 letters, whose LIS has length 980 is equal to 
the number whose LIS has length 21. 
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3 Avoiding the pattern (132) 

The case of (132)-avoiding permutations is handled by the following sequence of observa- 
tions. 

• The number f{n, k) of n-permutations that avoid (132) and whose longest increasing 
subsequence has length < k is equal to the number of ordered trees of n edges whose 
height is < k. This follows from a bijection of Jani and Rieper jTU]. 

• It is known from the theory of ordered trees ^ that this number is 



f{n,k)=2 



2n \ 1 / 2n + 2 
n + t{k + l)J ~ 4:\n + l + t{k + l) 



(A; = l,2,3,...). 

(3.1) 

To see that the normalized random variable 

LIS(j9„) - Vvm 



defined on (132)-avoiding permutations p„ of n letters, has a nontrivial limiting 
distribution, as n — oo, we consider 

Prob(X(p„) <e) = Prob(LIS(p„) < (0 + ^)^) 

f{n, V^(g + 0F)) 

Cn 

where 

1 /2n 



n + l\n 

is the nth Catalan number. But by ()3.H) we see that 



Cn , 

I = — CXD 



in which the general t-th term of the sum is 



,/ X n + lf 2n \f (n+l)(2n + l)\ , , „ 

^("■"-7y(„+i,yi)(^- (Ui)"-.v.J - (^-0^^} 

It is an easy exercise in Stirling's formula to see that 

lim 0(n,t) = (1 - 2cV)e-"'*' 

n— »oo 

and the proof is complete. 
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The limiting cumulative distribution function F{9), in ()1.2|) . is related to the Jacobi 
theta function 

oo 
t=— oo 

In fact, F{e) = Q{u) + 2uQ'{u), where u = {1 + Qj^fiif. It is well known that this 
is indeed the limiting distribution, from the theory of ordered trees, but the derivation is 
simple enough that we have included it above. Indeed [Sj shows that this same distribution 
function, originally considered by Renyi and Szekeres ^H]; is the limiting distribution 
function for many classes of trees. 

We remark that although the Jani-Rieper bijection ^U] is the most explicit mapping 
between 132-avoiding permutations of n letters whose LIS has length k and ordered trees 
of n edges whose height is /c, there are other, somewhat less explicit, bijections in the 
literature. Indeed, in Krattenthaler ^T], we find an explicit bijection between such per- 
mutations and Dyck paths. This mapping has the property that the length of the LIS 
corresponds to the height of the image path. In view of well known bijections between 
Dyck paths by height and ordered trees by height, we have, by composition, the mapping 
that we need here. Likewise in Chow, West |4J there are remarks in section 5 that suggest, 
less explicitly, that such bijections exist. Other, equivalent bijections with Dyck paths 
have been given by Fulmek jU], Reifegerste ^3], and Bandlow and Killpatrick [S]. 



4 The (321) case and the Erdos-Szekeres theorem 

Finally we ask for the limiting distribution function for the length of the longest increasing 
subsequence in the class 5*^(321) of (321)-avoiding permutations of n letters. Let /(n, fc) 
denote the number of such permutations whose longest increasing subsequence has length 
<k. A famous theorem of Erdos-Szekeres states that every permutation of (r — l)(s — 1) + 1 
or more letters contains either a decreasing subsequence of length r or an increasing 
subsequence of length s. If we take r = 3 we see that every permutation in the class 
'S'2fc+i(321) has an increasing subsequence of length k + 1, i.e. /(n, fc) = for all n > 2k + l. 

Even though, for each fixed k, f{n, k) vanishes for all large enough n, the question of 
the limiting distribution function of the LIS remains. 

Theorem 4. The number a{n, k) of (321) -avoiding permutations having longest increasing 
subsequence of length = k is given by 

, f2k-n+l fn + l\V , , 

for [{n + l)/2j < k < n, and vanishes otherwise. 

These permutations correspond 1-1, under the Schensted insertion algorithm (see, e.g., 
[in]), to pairs of Young tableaux with n cells, at most two rows, and k columns, so the 
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number of them is B^, where B is the number of such tableaux. A brief computation 
with the hook formula shows that 

^ 2k-n + lfn+l 
B — 



n + 1 \n — k J ^ 

completing the proof. □ 

It follows that the number f{n,k) of permutations in 5'„(321) whose LIS has length 
< A; is 

2j — n + l/ n + l 



L(n+l)/2j<j<fc ^ \J ' / / 

Now fix t > 0. The probability that a permutation of n letters has LIS of length at most 
n/2 + t^/n is 

f(n, n/2 + t^/E) 

1 (2n\ 
n+1 V n / 

Hence the limiting probability distribution function that we seek is 

n+l\nJ [{n+l)/2}<j<n/2+t^ ^ 

4 x-^ ,o/ n+1 ^ ^ 



= lim jTT-r / k , , , , , . 

n-oo(n+l)P") ^ , \{n+l)2 + k 

By an easy application of Stirling's formula the relation 

/ 2m \ 

\m—k) —k'^/m 
\m ) 

holds uniformly for all \k\ < t\/rn^ and we have 



F(t) = ^lim^ y ^e-^'^Vn 
^ ^ o<fc<tV^ 

The quantity whose limit is being taken is a Riemann sum that approximates the integral 







with grid size 1 / y/n, and so the desired limit is 



F{t) = ^ f x''e-^"dx 
V^r Jo 

A Jo 

r(3/2,4t2) 

r(3/2) ■ 
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The limiting distribution F{t) is well-known in statistics: Except for a linear change of 
variables, it is a distribution with three degrees of freedom, which arises as the distri- 
bution of the length of a 3- vector whose entries are independent standard normal random 
variables (see, for example, ^). It would be interesting to find a heuristic explaining this 
connection. 

5 Other patterns of three letters 

We have so far dealt with the statistics of the length of the longest increasing subsequence 
in permutations that avoid any one of the patterns (132), (231), or (321). There remain 
three other patterns of three letters, viz. (123), (312), and (213). 

The (123) case is trivial. If an n-permutation avoids (123) then its LIS has length 
exactly 2 except for the reversal of the identity permutation, whose LIS has length 1. 

The (312) case is identical with the (231). Indeed, the length of the LIS of a (312)- 
avoiding permutation is equal to the length of the longest decreasing subsequence of 
its reversal, which is (213)-avoiding, and that is equal to the length of the LIS of the 
complement of its reversal, which is (231)-avoiding, as claimed. 

Similarly, by reversal and complementation, the (213) case is identical with the (132). 

We thank Christian Krattenthaler, Brendan McKay and Julian West for informative 
discussions related to this paper. 
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